fix(workflows): refresh pending email sender verification - #113091

Problem

Workflows users can finish DNS setup and still see their sender as unverified until they open its verification UI. After this PR, a new sender shows as verified within about a minute of SES accepting its DNS.

Impact

Pays into Grow Workflows through Workflows activation rate. A sender that verifies without the user on the page removes a return visit before the first send, and the upcoming "send once verified" change builds on this sweep so a waiting workflow can go live by itself. Expected move on its own: small, up.

Changes

• Background checks update the verified flag, so Channels and MCP reads pick it up without opening the setup UI.
• Senders get checked every minute for an hour after verification is requested, then every 5 minutes up to 7 days. Creating a sender, a pending verify click and a resolved advisor finding request verification. Senders without a request time fall back to created_at. Each tier has its own lock and cursor.
• A resolved SES VDM advisor finding for a domain, such as "DKIM record was not found", arrives through the existing SES events webhook. It requests verification for the pending senders on that domain, so the next 1-minute sweep checks them whatever their age. A failed check is retried like any other pending sender.
• Background checks read SES and DNS. They require successful domain verification, DKIM, matching MAIL FROM, DMARC, and the project's SES association.
• One SES status read covers up to 100 senders. Each kind of read, including tenant reads, runs at most once every two seconds, half the SES limit, so the verify button keeps headroom. Advisor events only write a timestamp, so a burst of events adds no parallel SES readers. User requests are not paced.
• The DNS and tenant checks run only for senders that SES already reports as ready.
• The sweeps skip senders whose last verification request is older than 7 days. SES fails a domain after 72 hours without DNS, and only the verify button restarts it.
• Background SES calls have connect and read timeouts and two attempts in total. A step starts only when its worst case, including retry backoff and pacing, fits the task's budget, so the task ends before its hard limit.
• MAIL FROM matching ignores case and a trailing dot.
• The verify button marks only senders on the same domain and MAIL FROM subdomain as verified. SES holds one MAIL FROM per domain, so other subdomains stay pending.
• Verification updates matching senders and reloads sending workers. It writes only the verified flag, preserving other configuration fields.
• Verified senders become the project's default when none exists, preserving #114682. The matching senders are locked while the flag and default change, so an edited sender cannot become the default.
• Provider failures leave the sender pending and allow the sweep to continue. Verified senders leave subsequent sweeps.
• Each sweep resumes a bounded pass over pending senders. New arrivals join the next pass, allowing older pending senders to retry.
• Mechanical changes register the task and update SES fixtures with their MAIL FROM domain.

Before:

flowchart LR
    UI[Open verification UI] --> Check[Check SES and DNS]
    Check --> Saved[Save verified flag]
    Saved --> Reads[UI and MCP reads]
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000;
    class UI,Reads phYellow;
    class Check phRed;
    class Saved phGray;

After:

flowchart LR
    Beat[Every minute, under 1 hour old] --> Guard[Acquire tier lease]
    Beat5[Every 5 minutes, 1 hour to 7 days old] --> Guard
    Guard --> Pending[Resume pass over the tier]
    Advisor[SES advisor finding resolved] --> Queue[Mark pending senders requested]
    Queue --> Guard
    Pending --> Batch[Read SES status for 100 senders]
    Batch --> Check[Check DNS and tenant for ready senders]
    Check --> Saved[Update matching verified flags]
    Saved --> Reads[UI and MCP reads]
    Saved --> Workers[Reload sending workers]
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000;
    class Beat,Beat5,Advisor,Reads phYellow;
    class Guard,Pending,Queue,Batch,Workers phBlue;
    class Check phRed;
    class Saved phGray;

How did you test this code?

Test seams: the background Celery task, integration reads used by API/MCP, and the project API's default sender. Tests fake AWS, DNS, and elapsed time.

Test rationale: The closest existing test, test_email_verify_updates_integration, exercises explicit verification and mocks the facade. It cannot catch a missing background refresh. The task/API cases cover verification, incomplete checks, concurrent edits, failure recovery, overlapping sweeps, and retry fairness. Extended cases preserve default selection after merging master and reject an edited sender when its sibling verifies.

The review round added cases for batched status reads, stale senders, MAIL FROM case, a budget run out by slow SES calls, and the verify button skipping senders on another MAIL FROM subdomain. Each case fails when its fix is reverted.

The tier round added age-boundary cases for both sweeps, a per-domain check that ignores age and domain case, and a webhook case where an identity named team-5.example.com reaches the sender check, not the tenant sync. Each fails when its rule is reverted.

Master refresh on 25af2dba: the touched-area suites (sender sweep, SES provider, SES integration, SES events webhook, TestEmailIntegration, scheduler, task registration, team workflows config) passed locally, 112 tests. test_ses_provider.py runs on its own, away from the known logging-setup clash below. One drift fix: the sweep tests no longer clear the per-process Team.workflows_config cache, because #118777 removed it. hogli product:lint workflows and hogli ci:preflight --strict passed. Repository-wide mypy was not rerun after the refresh.

assert response.json()["config"]["verified"] is True
E assert False is True

Overlap without lock: assert 2 == 1
Retry during new arrivals before fix: assert [False, True] == [True, True]
After fixes: 56 passed
Review round, each fix reverted on its own:
  no 7-day window       -> stale_sender fails
  exact MAIL FROM match -> mail_from_differs_in_case fails
  one read per sender   -> both interrupted-sweep cases fail
  no per-step budget    -> budget_exhausted fails
  all-subdomain marking -> test_email_verify_updates_all_other_integrations_with_same_domain fails
Review round suites: 458 + 40 passed
Tier round, each rule reverted on its own:
  1-hour tier covers 7 days    -> settled_sender_not_in_minute_sweep fails
  exact domain match on event  -> domain_check_verifies_pending_senders_on_that_domain_of_any_age fails
  no advisor dispatch          -> resolved_advisor_finding_checks_the_senders_on_that_identity fails
Tier round suites: 478 + 40 passed
QA swarm round 3 fixes, each reverted on its own:
  domain removed before its check -> advisor_domain_stays_queued_until_its_check_finishes fails
  unpaced tenant reads            -> tenant_reads_for_ready_senders_are_spaced_apart fails
Retry probe against a local HTTP server returning Throttling:
  old adaptive max_attempts=2: 3 attempts
  new standard total_max_attempts=2: 2 attempts, under 1s backoff
Round 3 suites: 503 + 40 passed. No automated test covers the row lock, because it needs two real concurrent transactions.
QA swarm round 4 fixes, each reverted on its own:
  verify click keeps old request time -> test_email_verify_returns_ses_result fails
  tiers ignore the request time       -> 4 tier and advisor cases fail
  user requests paced                 -> test_verify_email_domain_success fails
Final suites on ed71c30b: 505 + 40 passed, and 550 passed across the email model, domain connect and full workflows backend tests.
Explicit verification, scheduler, registration and project config: 25 passed
Backend compilation: passed
hogli build: no generation pipelines triggered by the 11 PR files
Repository-wide mypy: no issues found in 23384 source files

.codex/with-flox hogli test products/workflows/backend/test/test_email_sender_verification.py products/workflows/backend/test/test_ses_provider.py products/workflows/backend/test/test_email_integration_ses.py
.codex/with-flox hogli test posthog/api/test/test_integration.py::TestEmailIntegration posthog/tasks/test/test_scheduled.py posthog/tasks/test/test_task_registration.py products/workflows/backend/test/test_team_workflows_config.py
.codex/with-flox python -m compileall -q posthog/models/integration/email.py products/workflows/backend
.codex/with-flox hogli build
.codex/with-flox hogli product:lint workflows
.codex/with-flox uv run mypy --cache-fine-grained .
.codex/with-flox hogli ci:preflight --strict

No frontend code or layout changes. Not checked: real SES, public DNS propagation, production mail delivery, or a real VDM advisor event. The advisor event shape follows the SES EventBridge docs. Three TestGetIdentityIspMetrics cases fail when they share a process with posthog/tasks/test/test_task_registration.py, which re-runs Django's logging setup and disables existing loggers. This happens on master too and gets its own fix PR. These tests do not read event properties.

Release status

• No feature flag controls this change

Docs update

None. No existing document under docs/ covers sender verification.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Involvement: no grilling rounds or prototypes in this refresh; earlier rounds ran two QA swarms and three single reviews; proof is the local test output above.

Agent: T3 Code / Codex, gpt-6.1-sol. Review-round fixes: T3 Code / Claude Code, claude-opus-5-5.

Skills: tdd, writing-tests, hogli, running-ci-preflight, qa-swarm, pr, writing-pr-descriptions, merge-risk, writing-voice, unslop, principle-prove-it-works.

Two QA rounds fixed overlapping sweeps, ordered failure mocks, an insensitive overlap test, and retry starvation during continuous arrivals. All findings are fixed or rejected.

The deferred-import comment request was rejected under the user's explicit no-code-comments instruction. GPT-6.1 Sol reviewed correctness and risk; GPT-6 Astra covered fresh eyes and design after Opus returned 429. CodeRabbit CLI skipped because it is signed out.

The master refresh used a merge commit. A read-only Claude Opus 5.5 review found that a changed sender could become the default when a sibling verified. The fix requires the checked sender to match the verification update; a failing task/API case proves the guard. The same guard excludes a sender deleted during the provider check.

A QA swarm (GPT-6.1 Sol: correctness, risk, design; Claude Opus 5.5: fresh eyes) on the tier round found a default-sender race, a retry count of three, unpaced tenant reads, parallel advisor readers, and dropped unfinished domain checks. All are fixed. The narrow fix round then found that the Redis advisor queue added in round 3 could starve, drop events, and drop failed checks. The queue was replaced by a per-sender verification_requested_at timestamp that the tier sweeps already process. That also covers re-added and re-verified senders. Hardening the queue with event generations was the alternative.

A critical Claude Opus 5.5 review after the merge flagged unbatched SES reads, an unbounded sweep, unbounded SES calls, and case-sensitive MAIL FROM matching. All four are fixed. The read-only check moved into the workflows service, which removed the duplicate SES path on the integration model and the one-function facade module. Three lower-risk findings are listed as known gaps. soft_time_limit was rejected: Celery raises it as an Exception, which the per-sender guard would swallow. Client timeouts and the per-step budget bound the run instead.

SES sends no EventBridge event when a domain identity, DKIM or MAIL FROM becomes verified. A resolved VDM advisor finding is the closest signal, so it only triggers a fresh check. The sweeps stay as the source of truth.

GitHub CI requires maintainer approval; CI and patch coverage remain unchecked. GitHub denied the requested assignee, so the PR remains unassigned.

A Claude Opus 5.5 (claude-opus-5-5) refresh moved the PR onto current master as one commit. Master had added a default reply-to on senders (#115898), turned off SES feedback forwarding (#113335), and registered new workflows task modules. All three merged without semantic conflicts. The per-team-instance config load from #118777 needed one test setup change. Skills: modernize-pr, pr-description.

The duplicate search found #110516 (on-demand domain status reads) and #113059 (verification access scope). Both closed without merging, and no merged PR adds a background sweep. Fixtures contain invented domains, addresses and tokens.